A flexible I/O arbitration framework for netCDF-based big data processing workflows on high-end supercomputers
نویسندگان
چکیده
1College of Computer and Information Science, Southwest University of China, Chongqing, China 2State Key Laboratory for Novel Software Technology, Nanjing University, Jiangsu, China 3RIKENAdvanced Institute for Computational Science, Kobe, Japan 4Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL, USA Correspondence Jianwei Liao, College of Computer and Information Science, Southwest University of China, Tianshen RoadNo. 2, Beibei, Chongqing, China. Email: [email protected] Funding information MEXTs program for the Development; Improvement of Next Generation Ultra High-Speed Computer Systems; National Natural Science Foundation of China, Grant/Award Number: 61303038; Opening Project of State Key Laboratory for Novel Software Technology, Grant/Award Number: KFKT2016B05; RIKENAdvanced Institute for Computational Science through the HPCI SystemResearch project, Grant/Award Number: hp150019 Summary
منابع مشابه
Toward a General I/O Arbitration Framework for netCDF Based Big Data Processing
On the verge of the convergence between high performance computing (HPC) and Big Data processing, it has become increasingly prevalent to deploy large-scale data analytics workloads on high-end supercomputers. Such applications often come in the form of complex workflows with various different components, assimilating data from scientific simulations as well as from measurements streamed from s...
متن کاملFlexAnalytics: A Flexible Data Analytics Framework for Big Data Applications with I/O Performance Improvement
a r t i c l e i n f o a b s t r a c t Increasingly larger scale applications are generating an unprecedented amount of data. However, the increasing gap between computation and I/O capacity on High End Computing machines makes a severe bottleneck for data analysis. Instead of moving data from its source to the output storage, in-situ analytics processes output data while simulations are running...
متن کامل2016 Olympic Games on Twitter: Sentiment Analysis of Sports Fans Tweets using Big Data Framework
Big data analytics is one of the most important subjects in computer science. Today, due to the increasing expansion of Web technology, a large amount of data is available to researchers. Extracting information from these data is one of the requirements for many organizations and business centers. In recent years, the massive amount of Twitter's social networking data has become a platform for ...
متن کاملProgramming Visual and Script-based Big Data Analytics Workflows on Clouds
Data analysis applications often include large datasets and complex software systems in which multiple data processing tools are executed in a coordinated way. Data analysis workflows are effective in expressing task coordination and they can be designed through visualand script-based programming paradigms. The Data Mining Cloud Framework (DMCF) supports the design and scalable execution of dat...
متن کاملDynamic configuration and collaborative scheduling in supply chains based on scalable multi-agent architecture
Due to diversified and frequently changing demands from customers, technological advances and global competition, manufacturers rely on collaboration with their business partners to share costs, risks and expertise. How to take advantage of advancement of technologies to effectively support operations and create competitive advantage is critical for manufacturers to survive. To respond to these...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- Concurrency and Computation: Practice and Experience
دوره 29 شماره
صفحات -
تاریخ انتشار 2017